Social Interaction: Multimodal Conversation with Social Agents

including robotics, artificial life, and artificial ecosystems. We present a new approach to human-computer interaction, called so&Z interaction. Its main characteristics are summarized by the following three points. First, interactions are realized as multimodal (verbal and nonverbal) conversation using spoken language, facial expressions, and so on. Second, the conversants are a group of humans and social agents that are autonomous and social. Autonomy is an important property that allows agents to decide how to act in an ever-changing environment. Socialness is also an important property that allows agents to behave both cooperatively and collaboratively. Generally, conversation is a joint work and ill-structured. Its participants are required to be social as well as autonomous. Third, conversants often encounter communication mismatches (misunderstanding others’ intentions and beliefs) and fail to achieve their joint goals. The social agents, therefore, are always concerned with detecting communication mismatches. We realize a social agent that hears human-to-human conversation and informs what is causing the misunderstanding. It can also interact with humans by voice with facial displays and head (and eye) movement. However, is autonomy itself sufficient for social services? Although autonomy is vital to survive in the real world, it is only concerned with “self.” It is selfish by nature. It seems that it does not work well in human society, since it includes socially constructed artifacts such as laws, customs, culture. Social services provided by computer systems have to incorporate with these artifacts. Socialness is a higher-level concept defined above the concept of an individual, and is the style of interaction between the individuals in a group. Socialness can be applied to the interaction between humans and computers, and possibly to that between multiple computers. In this paper, we study socialness of conversational interaction between humans and computers. Conversation is no doubt, a social activity, especially when more than two participants are involved in it. However, conversation research to date has been biased to problem-solving. Question-answering systems are typical examples. All conversation research based on this view has the following features.